The present invention relates to a position and orientation determination apparatus, method, and a storage medium therefor for determining the position and orientation of a capture means before performing an image process to generate an augmented picture using an image in an unreal space such as computer graphics incorporated into the real space by measuring a parameter indicating the position and orientation of capture means (image pickup means) for capturing the real space with a method other than using a captured picture, and identifying information about a picture whose parameter has been captured. The present invention can also be applied to any means and method for determining the position and orientation of a capture device used for any purpose other than augmenteding the real space and another space image.
In an AR (augmented reality) system for augmenting the real space by virtual space such as computer graphics, etc., the position and orientation of a camera cannot be precisely measured in a method, for example, using a position and orientation sensor such as a magnetic sensor, etc., other than using a captured picture, when the real space is captured by the camera. Therefore, when a virtual object is superimposed on the real space based on the position and orientation of the camera measured by the sensor, a displayed image is often shifted from a desired position.
Therefore, in the conventional technology, some methods have been suggested to improve the precision of an external parameter (position and orientation of a camera) for capturing the real space. In one of the methods, a plurality of feature points (landmarks) whose positions are known in a three-dimensional array are arranged in the real space. Then, the external parameter of the camera is corrected using an error between the actual position of a target landmark, included in some landmarks captured by the camera and displayed on the display screen of the camera, and the position of the target landmark predicted based on the position and orientation of the camera at the time, acquired by a position and orientation sensor.
That is, a target landmark can be a marker such as a seal having the information of specific color or form attached to an object in the real space, or a feature point in a natural scene.
In another method, when there are a plurality of landmarks in the real space, using three (or one or two) landmarks in the image captured by a camera, and a measurement value measured by a position and orientation sensor, a projective matrix from the real space (three-dimensional space) to the display screen (two-dimensional plane) of the camera is calculated by a matrix operation, and an external parameter of the camera is corrected using the obtained matrix.
However, in the former method of the above mentioned conventional methods, the external parameter of the camera is corrected based on the target landmark. Therefore, for example, when a virtual object is superimposed on the position at a distance from a target landmark, the object can be shifted from a desired position when it is displayed. Furthermore, since a target landmark is switched and changed when the view of the camera is changed, a correction value can be largely changed when it is switched.
On the other hand, in the latter method of the above mentioned conventional methods, an obtained projective matrix can cause an image to be mapped with incorrect orthogonality of the coordinate axis of the original coordinate system. For example, space distortion can occur depending on the detection precision of a landmark.
The present invention has been developed to solve the above mentioned problem, and aims at correcting the parameter indicating the position and orientation of a camera by reflecting a captured landmark.
To attain the purpose of the present invention, for example, a position and orientation determination apparatus according to the present invention has the following configuration.
That is, the position and orientation determination apparatus which identifies a parameter indicating the position and orientation of capture means for capturing a picture in a real space containing a plurality of feature points whose positions are known in a three-dimensional array comprising: position and orientation measurement means for measuring the position and orientation of the capture means in a method other than using a captured picture; detection means for detecting the plurality of feature points and their positions in the two-dimensional array on the image pickup screen using the picture in the real space captured by the capture means; prediction means for predicting the positions of the feature points in the two-dimensional array on the image pickup screen based on the position and orientation of the capture means measured by the position and orientation measurement means; and correction means for correcting the parameter indicating the position and orientation of the capture means based on the positions of the feature points on the image pickup screen of the capture means obtained by the prediction means, and based on the position of the feature points obtained by the detection means. With the configuration, the parameter indicating the position and orientation of the capture means for capturing the real space is identified by the correction means correcting the parameter.
Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.